REMARKS 

This application has been reviewed in light of the Office Action dated 
February 8, 2006. Claims 3-7 and 10 are presented for examination, of which Claims 3 and 
10 are in independent form. Claim 6 has been amended to define still more clearly what 
Applicants regard as their invention, and Claims 4 and 10 have been amended as to minor 
matters of form. Claim 13 has been canceled without prejudice or disclaimer of subject 
matter; it is noted that cancellation of this claim obviates the objection to the drawing. 
Claim 13 will not be mentioned further. Favorable reconsideration is requested.. 

A substitute specification is submitted herewith in both a marked and a 
clean version, to address the objection to the title, to update the Summary section, and to 
correct various minor informalities noted therein; no new matter has been added. 

Also, Applicants note the objection to the wording of Claim 6. That claim 
has been amended to clarify that a table, such as that shown in Fig. 7, defines band numbers 
and number of band information which correspond to a configuration of the spectral 
distribution data and are necessary to estimate multi spectral distribution data of color data 
in an image sensing device 101 (page 16, lines 19-23 of the specification).- The re- 
wording of that claim is believed to obviate that objection, the withdrawal of which is 
therefore respectfully requested. 

Claims 3-7 and 10 were rejected under 35 U.S.C. § 102(e) as being 
anticipated by U.S. Patent 6,980,231 (Ohsawa). 

Independent Claim 3 is directed to an image processing apparatus that 
comprises first and second acquisition sections, the first being arranged to acquire color 

- It is of course to be understood that the claim scope is not limited by the details of this or 
any other particular embodiment that may be referred to. 
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data of an object, and the second, to acquire spectral distribution data, which is necessary to 
estimate spectral distribution data of a total wavelength region, from a plurality of spectral 
distribution data in accordance with the acquired color data. Also provided is an estimator, 
arranged to estimate the spectral distribution data of the total wavelength region on the 
basis of the spectral distribution data acquired by the second acquisition section. 

Applicants emphasize that among other notable features of the apparatus of 
Claim 3, is that when spectral distribution data of a total wave length region is estimated, 
the estimation is made using spectral distribution data acquired from a plurality of spectral 
distribution data in accordance with color data acquired from an object. This means that 
the plurality of spectral distribution data is unnecessary for the estimation of the spectral 
distribution data of the total wavelength region in the apparatus of Claim 3. As a result, the 
image sensing time of a multi-spectrum camera, the data amount, and the calculation time 
required for estimating a spectral distribution are reduced. 

Ohsawa relates to a system in which a first digital camera 3 that acquires 
RGB image data of an object 1 . A spectral reflectance calculation unit 4 calculates spectral 
reflectance image data F corresponding to the RGB image data, and a second digital camera 
8 acquires image data P' of a color chip 6 under observation illumination 7. The color chip 
6 is formed by a plurality of unit color chips, each of which represents an independent 
spectral reflectance. A tristimulus value calculation unit 5 inputs the spectral reflectance 
image data F and the color chip image data P', and calculates tristimulus values XYZ 
corresponding to the spectral reflectance image data F under the observation illumination 7. 

The Office Action asserts that the recited second acquisition section 
corresponds to the digital camera 8, and the acquisition of spectral distribution data, which 
is necessary to estimate spectral distribution data of a total wavelength region, from a 
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plurality of spectral distribution data in accordance with acquired color data, is alleged to 
correspond to the description at col.5, lines 44-54, and col.5, line 60, to col. 6, line 30, of 
Ohsawa. Applicants respectfully traverse this interpretation of Ohsawa. In that patent, the 
spectral reflectance image data F and the color chip image data P' of the color chip 6 are 
acquired independently. Applicants note that the color data referred to in Claim 3 
corresponds to the spectral reflectance image data F based on the Examiner's opinion. 
Even if Ohsawa is deemed to teach that the tristimulus values XYZ are calculated from the 
spectral reflectance image data F and the color chip image data P\ this does not amount to 
a teaching or suggestion to acquire the color chip image data P' (spectral distribution data) 
in accordance with the acquired spectral reflectance image data F (color data). That is, 
Ohsawa does not teach the estimation of the spectral distribution data of the total 
wavelength region by acquiring partial spectral distribution data in accordance with the 
acquired color data as is recited in Claim 3. Accordingly, Claim 3 is believed to be 
allowable over Ohsawa. 

Independent Claim 10 is a method claim corresponding to apparatus Claim 
3, and is believed to be patentable for at least the same reasons as discussed above in 
connection with Claim 3. 

A review of the other art of record has failed to reveal anything which, in 
Applicants' opinion, would remedy the deficiencies of the art discussed above, as references 
against the independent claims herein. Those claims are therefore believed patentable over 
the art of record. 

The other claims in this application are each dependent from Claim 3, and 
are therefore believed patentable for the same reasons. Since each dependent claim is also 
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deemed to define an additional aspect of the invention, however, the individual 
reconsideration of the patentability of each on its own merits is respectfully requested. 

In view of the foregoing amendments and remarks, Applicants respectfully 
request favorable reconsideration and early passage to issue of the present application. 

Applicants' undersigned attorney may be reached in our New York Office 
by telephone at (212) 218-2100. All correspondence should continue to be directed to our 
address listed below. 



Respectfully submitted, 

Leonard P. Diana 
Attorney for Applicants 
Registration No. 29,296 

FITZPATRICK, CELLA, HARPER & SCINTO 
30 Rockefeller Plaza 
New York, New York 10112-3801 
Facsimile: (212)218-2200 

NY_MAIN568160v1 
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TITLE 

IMAGE PROCESSING APPARATUS AND METHOD THAT USE SAMPLE 
SPECTRAL DISTRIBUTION DATA SELECTED BASED ON COLOR DATA 
FROM AN OBJECT TO ESTIMATE SPECTRAL DISTRIBUTION OF A 

WAVELENGTH REGION 



FIELD OF THE INVENTION 
[0001] The present invention relates to an image processing apparatus and 
method and, more particularly, to an image processing apparatus and method 
which accurately estimate the display color of an acquired object, regardless of 
the conditions of observation illuminating light, by using the spectral distribution 
data of the object. 

BACKGROUND OF THE INVENTION 
[0002] The method most often used in apparatuses which acquire spectral 
distribution data is to divide into bands a wavelength region (to be referred to as a 
"total wavelength region" hereinafter) to be measured such as a visible light 
region, acquire the spectral distribution information of each band, and acquire the 
spectral distribution information of the total wavelength region from the results. 
[0003] FIG. 1 is a block diagram showing the arrangement of a multi-spectrum 
camera 207 for acquiring spectral distribution data. 
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[0004] A plurality of filters having spectral characteristics different in 
wavelength are arranged along the periphery of a disk-like rotary filter 202 in 
order of band wavelength. Reflected light from an object to be sensed passes 
through an optical system 201 and through one filter on the rotary filter 202, and 
is decomposed into light indicative of band information. By rotating the rotary 
filter 202 placed before a CCD sensor 204 by a driving motor 203, filters facing 
this CCD sensor 204 are switched, and input light is decomposed into light 
indicating each set of band information (that is, the information for each band) . 
[0005] The intensity of the light indicating band information is converted into a 
digital signal value by the CCD sensor 204 and an A/D converter 205, and stored 
as data in units of pixels into a data storage unit 206. 

[0006] Generally, the smaller the number of filters of the rotary filter 202, the 
shorter the processing time required for sampling and the higher the speed of 
image sensing. Therefore, the number of filters is minimized. 
[0007] As indicated by the solid lines in FIG. 2, the spectral distribution data 
acquired by the multi-spectrum camera 207 is the information of each band 
decomposed by the corresponding filter. Hence, this data cannot be directly used 
as the spectral distribution data of the total wavelength region. Therefore, it is 
necessary to acquire the spectral distribution data of the total wavelength region 
by interpolating the information of each band. Note that the spectral distribution 
data of the total wavelength region will be called "multi--spectral distribution 
data", with respect to band information acquired by the multi-spectrum camera 
207. 

[0008] In this interpolation, as shown in FIG. 2, an intensity value corresponding 
to the intermediate wavelength of each band is defined as band information 

(indicated by • in FIG. 2). These pieces of band information are connected by a 

curve (indicated by the broken line in FIG. 2) to obtain multi-spectral 
distribution data as the spectrum interpolation result. FIG. 2 shows the spectral 
characteristic acquired by the multi-spectrum camera 207 having six filters. In 
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this case, mult i -spectral distribution data must be estimated by using band 
information having six fixed filter characteristics. 

[0009] Methods other than the above interpolation process are available by 
which the spectral distribution data of the total wavelength region is obtained by 
using band information. One method will be explained below. 
[00101 A function for obtaining an output value at a wavelength X of the spectral 
distribution data information of each decomposed band is defined as Ln(X) where 
n=l, 2, . . . n corresponds to the information of each decomposed band. Spectral 
distribution data R(X) of the total wavelength region is calculated as a linear sum 
at different wavelengths of Ln(A,) given by 

y?(A) = ^{al.Ll(A) + a2-i:2(A) + ... + a«-LA2(^)} ... (1) 

A. 

[0011] where al, a2, . . . , an are arbitrary coefficients which can be different 
from one wavelength X to another 

[0012] In addition to the above method, methods represented by the KL 
expanding method are known by which the spectral distribution data of the total 
wavelength region is obtained from band information. Accordingly, an optimum 
method is properly used in accordance with, e.g., the characteristics of band 
information. 

[0013] A technique which realizes perfect color matching between environments 
differing in environmental illuminating light, i.e., environments having 
illuminating lights different in spectral distribution is disclosed in, e.g., Japanese 
Patent Laid-Open No. 9-172649. Since this technique uses spectral distribution 
data, it is necessary to acquire the multi spectral distribution data of illuminating 
light by using the multi-spectrum camera 207 as shown in FIG. 1, i.e., to estimate 
the spectral distribution of illuminating light. 

[0014] Estimating the spectral distribution requires not only the image sensing 
time of the multi-spectrum camera 207 but also, in order to process all band 
information acquired, large amounts of memories for storing all band information 
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in units of pixels and working buffer memories. In addition, a processing time for 
processing all the band information is necessary. 

[0015] Furthermore, to avoid errors produced by the interpolation process 
described above, it is necessary to use complicated interpolation and increase the 
number of band information. This increases the load (time) of image sensing and 
processing. 

SUMMARY OF THE INVENTION 
[0016] The present invention has been made to individually or collectively solve 
the above problems, and has as its object to reduce the load (time) of image 
sensing and processing when estimating a spectral distribution. 
[0017] To achieve the above object, one preferred aspect of the present invention 
discloses an image processing apparatus c omprising: a first acquisition section, 
arrang e d to acquire color data of an object; a second acquisition section, arranged 
to acquir e a plurality of spectral distribution data on the basis of the configuration 
of sp e ctral distribution data defined in accordance with the acquir e d color data; 
and a g e nerator, arrang e d to gen e rat e imag e data constructed by the acquired 
color data and the plurality of spectral distribution dat a that comprises first and 
second acquisition sections, the first being arranged to acquire color data of an 
object, and the second, to acquire spectral distribution data, which is necessary to 
estimate spectral distribution data of a total wavelength region, from a plurality of 
spectral distribution data in accordance with the acquired color data. Also 
provided is an estimator, arranged to estimate the spectral distribution data of the 
total wavelength region on the basis of the spectral distribution data acquired by 
the second acquisition section. 

[001 8 ] Anoth e r preferr e d asp e ct of th e pr e s e nt invention di s cl ose s an imag e 
pr o c es sing apparatus comprising: an input s e ction, arranged to input imag e data 
construct e d by color data and a plurality of s p e ctral distribution data; and an 
estimator, arrang e d to estimate sp e ctral distribution data of a total wavel e ngth 
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r e gion on th e basi s of th e configuration of the sp e ctral distribution data d e fin e d in 
accordanc e with the color data. 

[0019] Still anoth e r preferred aspect of th e present inv e ntion discloses an imag e 
proc es sing apparatus comprising: a first acquisition section, arrang e d to acquir e 
color data of an object; a s e cond acquisition s e ction, arrang e d to acquir e a 
plurality of spectral distribution data on th e basis of th e configuration of sp e ctral 
distribution data defin e d in accordance with th e acquired color data; an e stimator, 
arrang e d to e stimat e sp e ctral distribution data of a total wav e l e ngth region on th e 
basis of the configuration of th e spectral distribution data; and a generator, 
arrang e d to generat e th e sp e ctral distribution data of the total wavel e ngth r e gion 
from the acquired color data and the plurality of spectral distribution data, on th e 
basis of th e e stimated spectral distribution data of th e total wavel e ngth region. 
[0018] Other features and advantages of the present invention will be apparent 
from the following description taken in conjunction with the accompanying 
drawings, in which like reference characters designate the same or similar parts 
throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0019] FIG. 1 is a block diagram showing the arrangement of a multi-spectrum 
camera for acquiring spectral distribution data; 

[0020] FIG. 2 is a graph for explaining the spectral distribution data acquired by 
the multi-spectrum camera; 

[0021] FIG. 3 is a block diagram showing the arrangements of an image sensing 
device for sensing an object and an image processing apparatus; 
[0022] FIG. 4 is a graph showing the acquisition result of eight pieees^ sets of 
band information when a bluish color is sensed by the multi-spectrum camera; 
[0023] FIG. 5 is a graph showing a method of estimating multi-spectral 
distribution data in an embodiment; 

[0024] FIG. 6 is a graph for explaining band information necessary to estimate 
multi spectral distribution data specifying a skin color; 
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[0025] FIG. 7 is a view for explaining a look-up table stored in a band count 
value storage; 

[0026] FIG. 8 is a view for explaining a look-up table stored in a band number 
storage; 

[0027] FIG. 9 is a block diagram showing detailed functions of the image 
sensing device; 

[0028] FIG. 10 is a flow chart showing the image sensing procedure of the image 
sensing device; 

[0029] FIG. 1 1 is a block diagram showing detailed functions of the image 
processing apparatus; 

[0030] FIG. 12 is a flow chart showing the process procedure of the image 
processing apparatus; 

[0031] FIG. 13 is a view showing the relationship between multi-spectral 
distribution data and frame band information; 

[0032] FIG. 14 is a graph for explaining the process of compressing band 
information; 

[0033] FIG. 15 is a graph showing the spectral characteristic data of RGB; and 
[0034] FIG. 16 is a graph showing a color matching function in the XYZ 
colorimetric system. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
[0035] An image processing apparatus of an embodiment according to the 
present invention will be described in detail below with reference to the 
accompanying drawings. 
[0036] [Arrangement] 

[0037] FIG. 3 is a block diagram showing the arrangements of an image sensing 
device 101 for sensing an image of an object and an image processing apparatus 
104. 

[0038] The image sensing device 101 comprises a color data sensing unit 102 for 
acquiring the color data of each pixel of an input image, and a multi-spectrum 
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camera sensing unit 103 for acquiring that spectral distribution data of each pixel 
of the input image, which is defined by the acquired color data. Note that the 
color data sensing unit 102 can also be a color data sensing device connected to 
the multi-spectrum camera. 

[0039] Note also that the multi-spectrum camera sensing unit 103 can be a multi- 
spectrum image sensing device as follows. That is, filters for acquiring band 
spectral distribution data differing in wavelength are used in image sensing by the 
above multi-spectrum camera, such that all of these filters are used to cover the 
total wavelength region, thereby always acquiring the spectral distribution data of 
all pixels. When this data is stored from the image sensing device 101 is stored 
into a data storage unit 105, that spectral distribution data of each pixel of the 
input image, which is defined by the acquired color data,, is stored, and the 
subsequent image processing is performed. 

[0040] Furthermore, as will be described in detail later, the multi-spectrum 
camera can also include dedicated filters for previously defined specific colors. If 
it is determined that a previously defined specific color exists in acquired color 
data, a dedicated filter for the- that specific color found is used te-additionally to 
sense all pixels. The multi-spectrum camera sensing unit 103 can also be a multi- 
spectrum image sensing device like this. 

[0041] The arrangement of the multi-spectrum camera sensing unit 103 can also 
be a multi-spectrum image sensing device as follows. That is, of filters of the 
multi-spectrum camera, all previously defined filters are used to always acquire 
the spectral distribution data of all pixels. When this data is stor e d from the image 
sensing device 101 is stored into the data storage unit 105, that spectral 
distribution data of each pixel of the input image, which is defined by the 
acquired color data, is stored, and the subsequent image processing is performed. 
[0042] The output image signal data (the color data and the spectral distribution 
data defined by this color data) from the image sensing device 101 is stored in the 
data storage unit 105 of the image processing apparatus 104. 
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[0043] A color data look-up unit 107 stores information for forming main 
component spectral distribution data required by a main component spectral 
distribution data generator 108 to estimate multi spectral distribution data by 
referring to the acquired color data and the spectral distribution data. 
[0044] A multi-spectral distribution data estimator 109 estimates the multi 
spectral distribution data of each pixel from the generated main component 
spectral distribution data and/or the spectral distribution data defined by the 
acquired color data. 

[0045] The color data look-up unit 107, the main component spectral distribution 
data generator 108, and the multi-spectral distribution data estimator 109 
construct a multi spectral distribution data estimation processor 106. 
[0046] A color converter 1 10 converts the color of the multi-spectral 
distribution data of each desired pixel by using the estimated multi-spectral 
distribution data. A color data converter 1 1 1 converts the color-converted, multi z 
spectral distribution data of each pixel into color data used in an output device 
112. 

[0047] This output device 112 outputs an image by using the output color data 
from the image processing apparatus 104. 
[0048] [Multi-Spectral Distribution Data Estimation Method] 
[0049] This embodiment notes that pieees- sets of band information necessary to 
estimate spectral distribution data and the number of these pieces -sets of band 
information are specified for a certain arbitrary color. That is, the multi-spectrum 
camera senses spectral distribution data in accordance with a color specified by 
color data in an input image. Spectral distribution data is estimated by pieces sets 
of arbitrary band information specified by the color data value of the sensed 
spectral distribution data and the number of these pieces of arbitrary band 
information. In this estimation, pi e ces sets of band information and the number of 
these pieees- sets of band information are varied in accordance with the color data 
value. 
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[0050J Also, filters of the multi-spectrum camera are prepared for all bands 
necessary to estimate multi-spectral distribution data for all color data. For 
example, FIG. 4 is a view showing the result of acquisition of eight pi e ce s sets of 
band information when a bluish color is sensed by the multi-spectrum camera. As 
shown in FIG. 4, if the spectral distribution of input light is offset to a specific 
wavelength region, the multi—spectral distribution data is estimated by 
interpolating four or five pi e c e s sets of band information of the acquired band 
information, and the remaining four or three pieees -sets of band information are 
redundant. 

[0051] In this embodiment, the above offset of the spectral distribution is taken 
into consideration. Therefore, if input light is a bluish color, as shown in FIG. 5, 
only three pieees- sets of band information are acquired, and the multi— spectral 
distribution data of the input light is estimated by interpolation. Consequently, as 
shown in FIGS. 4 and 5, it is possible to obtain substantially the same estimation 
result and reduce the number of sets of band information. 
[0052] A method of estimating the multi-spectral distribution data from band 
information other than the above interpolation method is also available. As 
described in, e.g., "New Version of Color Science Handbook 2nd ed.", page 
1 196, FIG. 28.29(a), band information required to estimate the multi--spectral 
distribution data specifying a skin color need only be first to third sets of spectral 
distribution data as shown in FIG. 6. 

[0053] If color data acquired from an image to be sensed is color data 
representing a skin color, filters corresponding to those three pieees- sets of band 
information shown in FIG. 6, which are designated by the color data,, are selected 
from a plurality of filters of the multi-spectrum camera. By sensing the image by 
using each filter and acquiring these three pieees -sets of band information, multi- 
spectral distribution data corresponding to the color data of a skin color can be 
estimated. 

[0054] One method of estimating multi z -spectral distribution data R(X) 
corresponding to the color data of a skin color is to use a linear sum. 
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[0055] In this method, the above-mentioned three preces -sets of band 
information for the multi-spectral distribution data R<X) corresponding to the 
color data of a skin color are defined as first band information L1(X), second band 
information L2(A,), and third band information L3(T), which are functions 
concerning the wavelength X. The multi spectral distribution data R(X) 
corresponding to the color data of a skin color can be estimated by 

R(X) = £{al • L\(X)+a2 ■ L2(X) + a3 - L3(X)} ... (2) 

[0056] where al, a2, and a3 are arbitrary coefficients which can be different 
from one wavelength X to another. 

[0057] Compared to the band information different in wavelength required to 
estimate the multi-spectral distribution data of a bluish color, the wavelength 
regions of band information sometimes overlap., as in the case of the three pieces 
of band information for estimating the multi-spectral distribution data 
corresponding to a skin color. These three pi e ces sets of band information for 
estimating the multi— spectral distribution data corresponding to a skin color are 
also three main component spectral distribution data for estimating the multi- 
spectral distribution data corresponding to a skin color. 
[0058] Similar to the estimation of the multi-spectral distribution data of a 
bluish color by the interpolation method or the linear sum method, this main 
component spectral distribution data can be obtained by combining arbitrary band 
information differing in wavelength. 

[0059] Accordingly, te 4n order accurately to sense a specific color of high 
frequency, such as a skin color, it is better not to estimate the multi— spectral 
distribution data from a large number of pieees -sets of band information different 
in wavelength by using the interpolation method. That is, it is preferable to 
perform image sensing by the multi-spectrum camera by using a dedicated filter 
corresponding to a specific color as described above, acquire the main component 
spectral distribution data, and then estimate the multi-spectral distribution data. 
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In this way, it is possible to reduce the number of filters needed for use-in-image 
sensing and obtain good multi-spectral distribution data estimation results. 
[0060] In the arrangement of the multi-spectrum camera of the image sensing 
device 101 according to this embodiment, filter characteristics for use in multi- 
spectrum camera image sensing have a plurality of band characteristics different 
in wavelength. The camera is also equipped with dedicated filters for previously 
defined specific colors. By giving band information to these dedicated filters, it is 
possible te-equally to handle the filters having band characteristics different in 
wavelength and these dedicated filters for previously defined specific colors. 
[0061] Accordingly, in the selection of filters for multi-spectrum camera image 
sensing by the image sensing device 101 or in the selection of band information 
when image data is stored from the image sensing device 101 is stored into the 
data storage unit 105, necessary sets of band information are made to correspond 
to color data beforehand, and band information designated by acquired color data 
is selected. 

[0062] [Operation of Image Sensing Device] 

[0063] On the rotary filter of the multi-spectrum camera sensing unit 103, filters 
corresponding to band information necessary to estimate the multi-spectral 
distribution data of color data are arranged in accordance with band numbers. 
Also, for color data (for a color designated by color data) sensed by the color data 
sensing unit 102, sets of band information necessary to estimate the multi z 
spectral distribution data are defined beforehand. By designating a band number, 
the defined band information can be acquired by using a filter corresponding to 
the band number. 

[0064] That is, a look-up table as shown in FIG. 7 defines the band numbers and 
the number of sets of band information necessary to estimate the multi-spectral 
distribution data of color data in the image sensing device 101. 
[0065] Also, a look-up table as shown in FIG. 8 defines those filter numbers of 
the multi-spectrum camera sensing unit 103, which correspond to the band 
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numbers in the image sensing device 101, and the spectral distribution 
characteristics of these filters. 

[0066] FIG. 9 is a block diagram showing detailed functions of the image 
sensing device 101. FIG. 10 is a flow chart showing the image sensing procedure 
of the image sensing unit 101. Note that the look-up tables shown in FIGS. 7 and 
8 are stored in a band count value storage 602 and a band number storage 603, 
respectively. 

[0067] In step S801, the color data sensing unit 102 acquires the color data of 
each pixel to be sensed, and the acquired color data of each pixel is stored in a 
data storage unit 607. 

[0068] In step S802, the number of sets of band information necessary to 
estimate the multi-spectral distribution data is acquired as a count value n from 
the band count value storage 602 by referring to the acquired color data, and is 
set in a counter unit 604. Subsequently, in step S803 a band number 
corresponding to a count value n-1 is acquired from the band number storage 
603. In step S804, the acquired band number is supplied to a filter selector 605 to 
select a filter corresponding to the band number. Consequently, the rotary filter 
rotates to set the selected filter in front of the CCD sensor of the multi-spectrum 
camera sensing unit 1 03 . 

[0069] In step S805, the multi-spectrum camera sensing unit 103 senses a 
spectrum. In step S806, the counter unit 604 decrements the count value n. In step 
S807, whether count value n>0 is checked, and the processing from step S803 to 
step S806 is repeated the same number of times as the number of the band 
information. 

[0070] After spectrum sensing is performed the same number of times as the 
number of band information, the flow advances to step S808, and the band 
information (spectral distribution data) equal in number to the bands 
corresponding to the acquired color data are stored as image data in the data 
storage unit 607. 
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[0071] The process shown in the flow chart of FIG. 10 is repetitively performed 
for all pixels. The acquired color data and the band information equal in number 
to the bands corresponding to the color data are output to the image processing 
apparatus 104. 

[0072] The image sensing device 101 can also perform the following processing. 
That is, in the steps after the acquisition of the color data in step S801, all filters 
of the multi-spectrum camera are used to acquire the spectral distribution data of 
all pixels. When this data i s stored from the image sensing device 101 is stored 
into the data storage unit 105, only that spectral distribution data of each pixel of 
an input image, which is defined by the acquired color data, is stored. After that, 
image processing using the spectral distribution data of each pixel stored in the 
data storage unit 105 is similarly performed. 
[0073] [Operation of Image Processing Apparatus] 

[0074] The processing of the image processing apparatus by which the multi- 
spectral distribution data of each pixel is estimated from the image data acquired 
by the image sensing unit 101 and output to the output device 1 12 will be 
described next. 

[0075] FIG. 1 1 is a block diagram showing detailed functions of the image 
processing apparatus 104. FIG. 12 is a flow chart showing the process procedure 
of this image processing apparatus 104. Note that of the input image data from 
the image sensing device 101, the color data of each pixel is stored in a color data 
storage 701 of the data storage unit 105, and the band information of each pixel is 
stored in a spectral distribution data characteristic output value storage 706 of the 
data storage unit 105. 

[0076] Note also that look-up tables as shown in FIGS. 7 and 8 are stored in a 
band count value storage 702 and a band number storage 703, respectively, of the 
color data look-up unit 107. These look-up tables can also be acquired from the 
band count value storage 602 and the band number storage 603 of the image 
sensing device 101 and stored in the color data look-up unit 107. 
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[0077] In step S901, the multi spectral distribution data estimation processor 106 
reads out the color data of a pixel of interest from the color data storage 701 . In 
step S902, the multi spectral distribution data estimation processor 106 refers to 
the color data and acquires from the band count value storage 702 a count value n 
as the number of band information required to estimate the multi-spectral 
distribution data. In step S903, the multi spectral distribution data estimation 
processor 1 06 acquires from the band number storage 703 a band number 
corresponding to the count value. 

[0078] On the basis of the acquired band number, the multi-spectral distribution 
data estimation processor 106 acquires the band spectral distribution data 
characteristic from a spectral distribution data characteristic storage 705 in step 
S904, and reads out the band information from the spectral distribution data 
characteristic output value storage 706 in step S905. 

[0079] In step S906, a counter unit 704 decrements the count value n. In step 
S907, a check is made as to whether count value n>0 is ch e cked , and the 
processing from step S903 to step S905 is repeated the same number of times as 
the number of the band information. 

[0080] Although not shown in the table shown in FIG. 7, spectral distribution 
data configuration information or the like for designating the generation of an 
arbitrary number of main component spectral distribution data for estimating the 
multi spectral distribution data from the "band spectral distribution data 
characteristic and band information" with respect to the color data is defined 
beforehand. This spectral distribution data configuration information includes 
main component spectral distribution data generation designation information, 
the number of main component spectral distribution data, band configuration 
information necessary to generate each main component spectral distribution 
data, coefficient information necessary to generate each main component spectral 
distribution data, and each main component spectral distribution data for 
estimating the multi-spectral distribution data corresponding to the color data. 
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[0081] If in step S908 it is determined by referring to the color data that the 
generation of the main component spectral distribution data is not designated, the 
band spectral distribution data characteristics acquired in step S904 and the band 
information acquired in step S905 are supplied to the multi-spectral distribution 
data estimator 109. 

[0082] On the other hand, if in step S908 it is determined by referring to the 
color data that the generation of the main component spectral distribution data is 
designated, the band spectral distribution data characteristics acquired in step 
S904 and the band information acquired in step S905 are supplied to the main 
component spectral distribution data generator 108. 
[0083] In step S909, on the basis of the "band spectral distribution data 
characteristics and band information" defined by the color data, the main 
component spectral distribution data generator 108 generates the designated main 
component spectral distribution data for estimating the multi-spectral distribution 
data, by referring to the number of main component spectral distribution data, 
band configuration information necessary to generate each main component 
spectral distribution data, and coefficient information necessary to generate each 
main component spectral distribution data. Note that the main component spectral 
distribution data can be obtained by, e.g., interpolating the band spectral 
distribution data characteristics configuring the selected main component spectral 
distribution data defined by the color data. 

[0084] By using the "band spectral distribution data characteristics and band 
information" equal in number to the band information acquired in steps S904 and 
S905, or by using the spectral distribution data for estimating the multi-spectral 
distribution data corresponding to the color data including the main component 
spectral distribution data generated in step S909, in step S910 the multi-spectral 
distribution data estimator 109 estimates the multi spectral distribution data of a 
pixel of interest on the basis of the spectral distribution data for estimating the 
multi spectral distribution data corresponding to the color data including the 
generated main component spectral distribution data, or on the basis of the "band 
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spectral distribution data characteristics and band information". In step S91 1, the 
estimated multi-spectral distribution data is stored in the multi-spectral 
distribution data storage 709. Note that the multi— spectral distribution data can be 
estimated by using each main component spectral distribution data and/or the 
linear sum method using the band spectral distribution data characteristics and 
band information. 

[0085] The processing indicated by the flow chart in FIG. 12 is repeatedly 
performed for all pixels to calculate the multi— spectral distribution data of each 
pixel of the input image. This multi spectral distribution data is stored in a multi z 
spectral distribution data storage 709. 

[0086] By using that multi-spectral distribution data of each pixel of the input 
image, which is estimated by the above processing, the image processing 
apparatus 104 causes the color converter 1 10 to perform color conversion for the 
multi-spectral distribution data to obtain a desired output result, thereby 
obtaining the converted multi spectral distribution data. In addition, the color data 
converter 1 1 1 integrates the multi-spectral distribution data converted by the 
color converter 1 10, by a convolution operation using a color matching function 
of an appropriately selected colorimetric system, thereby converting into color 
vector data having three values. As this color vector data having three values, 
color space data unique to a device such as the L*a*b* colorimetric system or the 
XYZ colorimetric system is generally selected. FIG. 16 shows a color matching 
function in the XYZ colorimetric system, as an example of a color matching 
function. 

[0087] The obtained color vector data having three values is transmitted from the 
image processing apparatus 104 to the output device 112. The output device 1 12 
converts the color vector data having three values into output signal data, which 
is optimum in an observation environment, of this output device 112. 
[0088] Examples of the output device 1 12 are a display using an RGB signal 
values, a printer using CMYK signal values, and a display and printer using 
multicolor signal values other than RGB and CMYK. This output device has a 
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function, such as a color management system (CMS), for obtaining an 
appropriate output, by taking account of the output characteristics of the output 
device, with respect to the three-valued color data. Therefore, the color data 
converter 1 1 1 uses this CMS to generate an output signal appropriate for the 
output device 1 12. 

[0089] By the above processing, it is possible to set the bands and the number of 
the bands, corresponding to a color specified by the color data of an input image, 
of spectral distribution data to be sensed by the multi-spectrum camera, and to 
estimate the spectral distribution data from this band information. In addition, the 
spectral distribution data is estimated by changing the band information and the 
number of the band information in accordance with the color data value, thereby 
realizing image signal processing using multi--spectral distribution data. 
[0090] [Compression Process] 

[0091] When the color data and the band information are transmitted from the 
image sensing device 101 to the image processing apparatus 104, if a data 
compressor for compressing the band information is used to compress the data 
amount of the band information (image data) as small as possible and transfer the 
compressed data, the time of the data transfer can be shortened. A data expander 
for expanding the compressed band information is of course prepared in the 
image processing apparatus 104 to restore the compressed band information and 
store the restored band information in the data storage unit 105. 
[0092] The process of compressing image data having multi—spectral 
distribution data or compressing band information will be explained below. 
[0093] FIG. 13 is a view showing the relationship between multi-spectral 
distribution data and frame band information. To acquire the band information of 
each pixel, one frame band information is acquired for one wavelength. 
[0094] Noting the band information of a pixel of interest in each frame, 
differences between adjacent pieces of frame band information are obtained, and 
correlations (FIG. 14) between these pieces of frame band information are 
extracted. Next, differences between pixels are obtained in the band information 
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of one frame, and correlations between these pixels in the one-frame band 
information are extracted. On the basis of the correlations between the pieces of 
frame band information and the correlations between the pixels in the one-frame 
band information, optimum encoding is performed. Consequently, spectral 
distribution data can be efficiently compressed. 

[0095] Compared to the compression of the spectral distribution data described 
above, the compression of the spectral distribution data characteristics does not 
always require frame band information but need only use data of each 
wavelength. Therefore, differences between data of different wavelengths of the 
spectral distribution data characteristics are obtained and properly encoded. In 
this manner, the spectral distribution data characteristics can be efficiently 
compressed. 

[0096] In this embodiment as described above, the multi-spectral distribution 
data can be obtained by acquiring the spectral distribution band information by 
using a filter which is designated by the acquired color data of each pixel and has 
an arbitrary number of spectral distribution characteristics, without using all 
filters mounted on the multi-spectrum camera. Accordingly, it is possible to 
minimize the output data amount from the multi-spectrum camera and the load 
(time) of image sensing. 

[0097] In addition, in the image processing which outputs image data 
constructed by an input color data value and spectral distribution data, the 
configuration of the spectral distribution data is defined in accordance with the 
color data value. This makes it possible to estimate the spectral distribution data 
of an object by using a minimum spectral distribution band information 
configuration. Therefore, the processing load (processing time) of the image 
processing can be reduced. 

[0098] Furthermore, in image data constructed by an input color data value and 
spectral distribution data, the configuration of the spectral distribution data is 
defined in accordance with the color data value. Therefore, the spectral 
distribution data can be estimated by using minimum spectral distribution band 
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information corresponding to the color data. This makes it possible to reduce the 
processing load (processing time) of the estimation and provide an image 
processing apparatus which accurately reproduces the color of object. 
[0099] Modifications 

[0100] The spectral distribution data as output band information from the multi- 
spectrum camera includes output values of a plurality of wavelength data output 
values. In this embodiment, of an arbitrary number of wavelength data in the 
spectral distribution data, the spectral characteristic data of color data need only 
be obtained for at least three wavelength data. For example, three sets of 
wavelength data ef- from a plurality of sets of arbitrary wavelength data in the 
spectral distribution data can be RGB spectral characteristic data shown in FIG. 
15. Instead of this RGB spectral characteristic data, a data configuration directly 
having RGB values as color data can also be used. 

[0101] In the above embodiment, the acquisition of color data by a discrete 
sensor is explained as an example. However, it is also possible to obtain an 
output value having the spectral characteristic data of color data by the multi- 
spectrum camera, calculate the color data from the obtained output value, and 
perform the above processing for this calculated color data. 

[0102] This embodiment is not restricted to the use of the multi-spectrum camera 
using the rotary filter but can also be a multi-spectrum camera comprising an area 
sensor in which filters sensitive to a plurality of wavelengths are arranged. 
Furthermore, the embodiment can also be a multi-spectrum camera which 
switches a plurality of light sources each for emitting light having a specific 
wavelength and which comprises a line sensor in which filters sensitive to a 
plurality of wavelengths are switched. That is, the embodiment need only be a 
multi-spectrum camera capable of acquiring band information having 
predetermined spectral distribution data characteristics, regardless of the optical 
arrangement of the camera. 

[0103] As many apparently widely different embodiments of the present 
invention can be made without departing from the spirit and scope thereof, it is to 
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be understood that the invention is not limited to the specific embodiments 
thereof except as defined in the appended claims. 



NY Main 568546J.DOC 



